{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "2b30253f",
   "metadata": {},
   "source": [
    "本节的主要内容是对原始数据的预处理，通过pandas将原始数据转换为张量格式。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "118e17cf",
   "metadata": {},
   "source": [
    "### 2.2.1 读取数据集"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cb7b9c9f",
   "metadata": {},
   "source": [
    "创建CSV（逗号分隔值）文件，将数据写入CSV文件中"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "e91c99a3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "os.makedirs(os.path.join('..','data'), exist_ok = True) #创建文件夹\n",
    "data_file = os.path.join('..','data','house_tiny.csv') #创建CSV文件\n",
    "with open(data_file, 'w') as f:\n",
    "    f.write('NumRooms,Alley,Price\\n') #列名\n",
    "    f.write('NA,Pave,127500\\n') # 每行表示一个数据样本\n",
    "    f.write('2,NA,106000\\n')\n",
    "    f.write('4,NA,178100\\n')\n",
    "    f.write('NA,NA,140000\\n')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f845626",
   "metadata": {},
   "source": [
    "要读取CSV中的原始数据集，我们导入pandas包并调用read_csv函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "8c18a2da",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms Alley   Price\n",
      "0       NaN  Pave  127500\n",
      "1       2.0   NaN  106000\n",
      "2       4.0   NaN  178100\n",
      "3       NaN   NaN  140000\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv(data_file) #d ata_file是文件\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3693cb82",
   "metadata": {},
   "source": [
    "### 2.2.2 处理缺失值"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "65d31a05",
   "metadata": {},
   "source": [
    "NaN表示缺失值，为了处理缺失值，我们通常使用插值法和删除法。\n",
    "插值法：用一个替代之弥补缺失值\n",
    "删除法：直接忽略缺失值\n",
    "通过iloc，将data分成inputs和outputs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "ba04b781",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms Alley\n",
      "0       3.0  Pave\n",
      "1       2.0   NaN\n",
      "2       4.0   NaN\n",
      "3       3.0   NaN\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Lenovo\\AppData\\Local\\Temp/ipykernel_8588/4146302184.py:2: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  inputs = inputs.fillna(inputs.mean())\n"
     ]
    }
   ],
   "source": [
    "inputs,outputs = data.iloc[:,0:2], data.iloc[:,2]\n",
    "inputs = inputs.fillna(inputs.mean())\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6eac2397",
   "metadata": {},
   "source": [
    "对于inputs中的类别值或离散值，我们将\"NaN\"视为一个类别。pandas可以自动将此列转换为两列Alley_Pave和Alley_nan。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1a06a474",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   NumRooms  Alley_Pave  Alley_nan\n",
      "0       3.0           1          0\n",
      "1       2.0           0          1\n",
      "2       4.0           0          1\n",
      "3       3.0           0          1\n"
     ]
    }
   ],
   "source": [
    "inputs = pd.get_dummies(inputs, dummy_na = True)\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "caced96f",
   "metadata": {},
   "source": [
    "### 2.2.3 转换为张量格式"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d032d2e",
   "metadata": {},
   "source": [
    "现在inputs和outputs中所有条目都是数值类型，可以将他们转换成张量格式，进而用张量的相关函数做进一步操作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "fd46a165",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[3., 1., 0.],\n",
       "         [2., 0., 1.],\n",
       "         [4., 0., 1.],\n",
       "         [3., 0., 1.]], dtype=torch.float64),\n",
       " tensor([127500, 106000, 178100, 140000]))"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "x,y = torch.tensor(inputs.values), torch.tensor(outputs.values)\n",
    "x,y"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "096c4c5b",
   "metadata": {},
   "source": [
    "### 2.2.4 小结"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bf5fa6ab",
   "metadata": {},
   "source": [
    "pandas是数据分析工具，可以与张量兼容\n",
    "pandas处理缺失数据时，可以根据情况选择用插值法和删除法"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "393e1a49",
   "metadata": {},
   "source": [
    "### 2.2.5 练习"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff655a74",
   "metadata": {},
   "source": [
    "创建包含更多行和列的原始数据集\n",
    "1.删除缺失值最多的列\n",
    "2.将预处理后的数据转换为张量格式"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "a5f726da",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "data_file1 = os.path.join('..','data','homework1.csv') #创建CSV文件\n",
    "with open(data_file1, 'w') as f:\n",
    "    f.write('Values,Position,Ages,Salary\\n') #列名\n",
    "    f.write('NA,guard,23,127500\\n') # 每行表示一个数据样本\n",
    "    f.write('93,center,30,106000\\n')\n",
    "    f.write('79,NA,NA,178100\\n')\n",
    "    f.write('66,center,22,140000\\n')\n",
    "    f.write('84,guard,NA,23000\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "83720f0e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Values Position  Ages  Salary\n",
      "0     NaN    guard  23.0  127500\n",
      "1    93.0   center  30.0  106000\n",
      "2    79.0      NaN   NaN  178100\n",
      "3    66.0   center  22.0  140000\n",
      "4    84.0    guard   NaN   23000\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "data = pd.read_csv(data_file1)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "a2e3f004",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Values Position  Salary\n",
      "0     NaN    guard  127500\n",
      "1    93.0   center  106000\n",
      "2    79.0      NaN  178100\n",
      "3    66.0   center  140000\n",
      "4    84.0    guard   23000\n"
     ]
    }
   ],
   "source": [
    "def delete_col(m):\n",
    "    num = m.isna().sum() #对na进行统计累加\n",
    "    num_dict = num.to_dict() #转换成字典，可以到处na累计值最大对应的索引号\n",
    "    max_key = max(num_dict,key=num_dict.get) #通过max找到对应索引号\n",
    "    del m[max_key] #删除对应的列\n",
    "    return m\n",
    "delete_col(data)\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "407673ec",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Values Position\n",
      "0    80.5    guard\n",
      "1    93.0   center\n",
      "2    79.0      NaN\n",
      "3    66.0   center\n",
      "4    84.0    guard\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\Lenovo\\AppData\\Local\\Temp/ipykernel_8588/3506884636.py:2: FutureWarning: Dropping of nuisance columns in DataFrame reductions (with 'numeric_only=None') is deprecated; in a future version this will raise TypeError.  Select only valid columns before calling the reduction.\n",
      "  inputs = inputs.fillna(inputs.mean())\n"
     ]
    }
   ],
   "source": [
    "inputs,outputs = data.iloc[:,0:2],data.iloc[:,2]\n",
    "inputs = inputs.fillna(inputs.mean())\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "b62cd6c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   Values  Position_center  Position_guard  Position_nan\n",
      "0    80.5                0               1             0\n",
      "1    93.0                1               0             0\n",
      "2    79.0                0               0             1\n",
      "3    66.0                1               0             0\n",
      "4    84.0                0               1             0\n"
     ]
    }
   ],
   "source": [
    "inputs = pd.get_dummies(inputs, dummy_na = True)\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "e8cdcb5b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(tensor([[80.5000,  0.0000,  1.0000,  0.0000],\n",
       "         [93.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [79.0000,  0.0000,  0.0000,  1.0000],\n",
       "         [66.0000,  1.0000,  0.0000,  0.0000],\n",
       "         [84.0000,  0.0000,  1.0000,  0.0000]], dtype=torch.float64),\n",
       " tensor([127500, 106000, 178100, 140000,  23000]))"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import torch\n",
    "x, y = torch.tensor(inputs.values),torch.tensor(outputs.values)\n",
    "x,y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4e53ee4",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
